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SUMMARY 



This report provides a synopsis of a six-year research and development 
(R&D) effort to investigate the feasibility and desirability of using computer 
assistance to improve leadership and management practices at Recruit Training 
Commands (RTCs). Information has been derived on the suitability of two com- 
puter systems and various approaches for employing these systems in pursuit of 
this goal. These approaches encompass instruction and performance evaluation 
for Company Commanders (CCs) as well as more general information acquisition 
and processing capabilities to assist in the management of RTCs. 

_ Some of these programs are ready foi? operational use, while others would 
require consideinable development and testing prior to operational implementa- 
tion.. There is a variety of unique and important contributions (e.g., behav- 
ioral objectives, training strategies, analytic procedures) to be derived from 
each of the programs, regardless of their state of operational readiness. 
Further, a new slant on the concept concerning organizational development was 
introduced, and the foundations for its operational implementation have been 
laid. This concept involves the i ntroduction of a "self-correcting" feature 
into the organization. It appears that this feature is required for aeneral 
organizational effectiveness and especial ly for improvement .in the areas of 
leadership and management. An effort is now underway to make this feature a 
permanent part of RTC. 



Preliminary evidence is available tO'-Tndicate that, for the most part, 
RTC Orlando does have the capabilities and motivation required to properly 
use, maintain and extend the various components of Computerized Evaluation 
and Training System (CETS) and is proceeding vigorously to do so. This report 
presents some highly regarded projects, utilizing CETS, which have been 
planned and Initiated under RTC initiative and control since the termination 
of most Naval Training Equipment Center (NAVTRAEQUIPCEN) involvement in the 
project a couple of months ago. The extent to which "self-correcting" orien- 
tations, such as exhibited at RTC Orlando, persist or can be instituted where 
it is lacking 'is a subject for further investigation. The existence of proper 
user attitudes and orientations is especially critical to this project because 
the system, in its present state of development, still requires considerable • 
levels of creativity and skill to exploit many of its current and potential 
capabilities. These demanding personnel requirements should be reduced as 
further development and use of the system establishes routine procedures of 
operation and places greater burden for 'its operation on the computer. 

The evaluation and demonstration of training and cost-effectiveness is a 
perennial problem In this area of research, and it was a problem in this 
project. The informational programs and the associated "self-correcting" 
feature of CETS are viewed as a promising approach to remedying many of these 
problems. The present efforts at program development and evaluation, however, 
took place under more traditional conditions. In spite of the limitations 
inherent 1n the traditional approach, generally, favorable and encouraging 
results were obtained. 

The heed for a different R&D paradigm became evident when some important 
results were found to be inconsistent from one situation to another and cer- 
tain issues critical to the R&D could not be addressed using' traditional R&D 
procedures. All things considered, there appears to be adequate evidence to 
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merit further development and use of the GETS concept and associated products 
in the Navy« In this regard, a number of reconmendations are niade: 

/ ^ ■ 

a. The GETS capability should be implemented at one RTC/and then, if 
warranted, at the three RTGs. (This process would require assurance that the 
system is operationally ready and that the RTGs have the capability to proper- 
ly utilize it.) 

b. Once implemented, the self -evaluating and self-correcting capabili- 
ties of GETS should be utilized to continually improve the system. This can 
be done, for example, by correlating the various measurers which are intended 
-to be predictors of CC success "^with the CG success criterirn measures, and 
eliminating those variables which do not predict at an acceptable level. 
Also, new variables can be added to the system at any time^ and tested for 
their predictive validity. The self-eval nation process should be a Continuous 
one, since changes in the organization may alter the utility of the HcTS 
variables. 

c. Continued efforts to determine the best and most valid measures of 
training effectiveness should be given high priority,' and these measures 
should be included in the GETS system for evaluation and improvement of all 
aspects of organizational operations. " 

[ d. Alternate techniques for implementing and exploiting the present 
/programs should be investigated, as part of the operational R&D effort. 

e. Further laboratory investigations should be carried out to develop 
"^tnnevative programs for improving the quality of leadership/management actions. 

f. The various products and by-products of this project (e.g., training 
programs, training strategies, dovelopmental procedures, evaluation oroced- 
ures, performance evaluation approaches, behavioral objectives, etc.) should 
be considered for possible contributions to,' and integration with, other R&D 
efforts in the area. 

g. A coordinated tri-service R&D effort should be considered for this 
area wherejn common programs are developed for use across all segments of the 
military. 

h. To determine the feasibility, desirabil ity, and general nature of 
this tri-service effort, the conditions and requirements of all potential 
users of the programs should be identified and the nature/of all related 
ongoing and planned R&D and operational programs should be reviewed. 
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PREFACE 



Since Us initiation in early 1 973, this project has been funded at 
various times by the Defense Advanced Research Projects Agency, tfie Navy 
Personnel Research and Devel opment Center, and the Office of Naval Research. 

Original conceptions for the project were established by Dr. J?m Regan as 
a test of the PLATO IV computer-based system. The ,jroject has evolved since 
that time to include many objectives in addition to those originally defined. 

Many people, both mil itary and civilian, have contributed to this project 
over the years. In recent times, the outstanding efforts and attitudes of 
LCDR William Sullivan and CTRCS Paul Zetterholin of RTC Orlando have contrib- 
uted invaluably to the project. .Mr. Marty Smith, aided by Mr. Pat Smith, ex- 
tended and refined the complex computer programs that he developed in earlier 
efforts. Pat Smith additionally performed many of the .critical day-to-day ' 
tasks of system operation and eval ua-tion. Mr. George Romot continued to 
assure proper liaison between the NAVTRAEQUIPCEN and RTC in the sensitive and 
difficult areas of system implementation and acceptance. 

A one-iear field trial is being planned in which RTC will assume control 
of most of the operations of the system. NAVTRAEQUIPCEN support will be mainly 
in the area of software maintenance. This trial period will provide critical 
information on the viability of the present system at an RTC. Additionally, 
this trial period will indicate the desirability of performing further research 
and devel opnent to improve the system's viability. Information on the ability 
of RTC to effectively utilize the system was not available from earlier field 
tests because of the extensive involvement of NAVTRA.EQUIPCEN personnel. Also, 
some of the more desirable capabilities of the' system were not available for 
operational empl oyrri en t until recently. Future plans for the systbm will be 
based on the outcome of this field test. 
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SECTION L 
INTRODUCTION 

The management and leadership area is a major source of problems in 
organizations. Of particular concern. are the affective and interpersonal com- 
ponents of this area; because these components are critical to organizational 
success, and because they are so resistant to efforts to imprbve them. 

RTCs hshve an additional reason for Interest in this area. Not only are 
effective leadership and management practices critical to their mission, but a 
large part of their mission is to instill such practices in others. 

In view of tJte contribution of computers to other instructional areas, a 
series of studies was initiated in early 1973 to investigate the feasibility 
of using a computer to improve leadership and management practices at RTCs. 
Special emphasis was to be placed on the interpersonal and affective areas. 
Prior investigations in this area were practically nonexistant (with two ex- 
12 

ceptions * ), so it was necessary to begin the project by identifying a varie- 
ty of ways in which a computer might be applied to this area. Several of 
these potential -appl ications were developed and evaluated, providing a basis 
for further research and appl ication in the area. 

This approach to the project evolved from an originating plan to evaluate 

the capabilities of the PLATO IV computer-based education system? for im- 
proving the attitudes, and thereby the behaviors of CCs at RTCs. 'The original 
intention was to focus attention on attitudes, toward general concepts (e.g., 

job satisfaction^ respect for recruits). Prior research findings^'^, however, i 
indicated that the instruction would be better directed at attitudes toward 
the specific behaviors desired on the job. 

The original conception of the project was further modified to include 
training on the types of behaviors that a CC should perform. The idea of this 
approach was that, for certain kinds of GC behaviors, direct modification of 
the behaviors through instruction and practice is the best way to change both 
attitudes and job performance. . 
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In the process of pursuing these extended objective?, programs were de- 
veloped or suggested which appiiared to be of potential benefit to RTCs. Thus, 
a new phase of the project was initiated in FY 77 to develop a training and 
evaluation package (in accordance with the joint reHuirements of the RTC, 
Orlando, and the NAVTRAEQUIPCEN) around one of the previously developed 
programs, for adoption by RTC. 

Detailed descriptions and results of the R&D performed prior to FY 78 are 

presented in other reports^*^'^'^'^'^'^^ The current report presents^.^rf' 
overall view of the entire project. However, an emphasis is placeJ on the^"^ 
activities and results from the FY 78 effort to coriplete ,the development, im- 
plementation, and evaluation of the newly configured CETS that was^fiitiated 
in FY 77. ^ 



Spencer, G. J. and Hausser, D. L.; Blaiwes, A. S. and Weller, D. R. use 
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SECTION II 

PROJECT OVERVIEW: FY 73 - FY 78 

This section describes the results of efforts to meet th\^ major project 
objectives discussed in the foregoing. At a very general level, the major ob- 
jectives fall into the following categories: 

Objective A - Evaluate the PLATO IV system. 

Objective B - Develop, demonstrate, implement, and evaluate computer- 
based programs for instructing and evaluating CCs in the leadership and . 
management areas. 

OBJECTIVE/A - EVALUATE THE PLATO IV SYSTEM 

General experiences with the PLATO IV system have been documented in a 

variety of reports jhe most salient disacWantages of the system for this 
project's purposes were its costs of operation and its unreliability, "i/iese 
problems sc^em to be due to the total dependence of the system on long distance 
telephone lines; and perhaps to the developmental state of the system at the 
time it was. used for this project. 

In the interest of reducing these disadvantages, the PLATO IV systeni'vas 
replaced by a stand-alone minicomputer. This new system (described in- a 

previous report ) performed all of the essential tasks previous>/'perfomied 
by PLATO IV as well a:-; some additional tasks (e.g., card reader' input, video 
tape control) at a Icter overall cost. ® 

Certain features of the PLATO IV system coul d nptr^be dupl i/>€?&dyti th the 
new system. However, only minor use of these features was made, and their 
absence was not a serious \oss. One PLATO IV feature which was missed was the 
inter-user program access and communication c^jiabMity, which allowed a ready 
interchange of programs and ideas among users. Some desirable applications 
for this feature were conceived, and one^'as partially^developed and tested (a 
computer-moderated role play between two students). /The decision to terminate 
the use of PLATO IV precluded further pursuit of these ideas. 

Other PLATO IV features could not be duplicated on the na^ system (e.g., 
touch panel Input, selective screen erase), but satisfactory methods for oper- 
ating without these features were fo'-nd, and their absence caused no major 
problems. 

The absence of the TUTOR language of the PLATO IV system would have been 
a stTious problem, especially siriLe many programs had to be translated to the 
new system. However,, a new computer language was developed which translated 
miich of the TUTOR code to a_^form compatible with the new system. 

■■■ ■ ■ . ■ ■ ) V -■ ■ ■ .• ^ . ■ 



14. See footnotes 6 - 11 on page 8. 

15. See footnote ll on page 8. 
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The new system was superior to PLATO IV" when it came to the input and 
sto»^age of Targe amounts of data. The new system included a high speed opti- 
cal card reader which allowed rapid entry of questionnaire responses, and a 
ten megabyte disk system which allowed storage of all RTC data. All inputs 
to the PLATO IV system had to be made through keyboard, and data storage 
space allocated to each user was much smaller. 

Another fee.ture of the current system which '/as not available on PLATO IV 
was the au^tomatic control of a video-tape playar. . The new system, with its 
high speed printer, also made the production of hard copies of programs and - 
displays jujch easier (PLATO IV used a slower, wet-process copy machine). 

OBJ€€frvT B - DEVELOP, DEMONSTRATE, EVALUATE AND IMPLEh^ENT COMPUTER-BASED 
^f^OGRAMS. / 

This^ R&D contributes a variety of infoniiation and products to CC training 
and evaluation in the area of leadership and management, as well as to the 
leadt^rship and management areas in general. These contributions stem from the 
identification of behavioral objectives and training objectives, and extend to 
the operational -ready programs which are directed at these objectives. These 
programs are described in detail in the earlier project reports. These devel- 
l^^fnents provide a framework for training and evaluation in. these areas and 
cJffer useful and innovative procedures and materials for implementing the 

programs. / 

/ 

/ 

DEVELOP COMPUTER-BASED PROGRAMS. Seven major areas* of development were identi- 
fied in the various project reports. Four of these programs are designed to 
teach interpersonal dnd managerial skills by taking various job-oriented (simu- 
lation) approaches to the CCs job. Two other programs employ a Socratic 
approach to persuade the CC to act in certain ways. A final program provides 
evaluative and diagnostic information on the performance of CCs as well as 
^ serving as a general management tool for RTCs. 

a. Case Study Program.^ ^ One job-briented program takes the student 
through a "case study" in which the student selects a recruit chief petty 
officer and then assigns tasks to the company. The actions of the student are 
accomplished by selecting from a menu of choices which are used to gather in- 
formation, and take actions. The performance of the company^ improves or de- 
grades depending on the quality of the decision making activ^^ies of the stu- 
dents, just as on the job. \ 

b. Problem Situation Program. ^'^ The job-oriented characteristics of a 
second, program consist of "problem situations" wherein interpersonal situations 
likely to be encountered on the job are described along with alternative re- 
sponses commonly made by CCs. to these situations. The student tries to choose 
the response that best represents the concepts being taught. Feedback is pro- 
vided, tailored to the particular response selected, concerning why it is 



~1 6. See f ootno^tentr^rrrpa^^ST-— 

17... SeiB footnotes 6, 8, 10, H on page. 8. ~ ^ 
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correct or incorrect. Tutorial and drill and practice modes are employed to 
teach the concepts of interpersonal performance • (e.g. , gjiving feedback, setting 
goals) that apply in th^se situations. The job-oriented aspect of the ins/truc- 
tion is derived from the fact that, in contrast' with morfi theoretical approach- 
es, the program teaches job skills by rp' • >o thn to specific appli- 
cations in specif ic job situations. 

c. Video Problem Situations.. uroaches to the job- . 

oriented mode utilize videotape to • .udtions Ibetter and to permit 

more natural responses to the situations. One of these programs replaces the 
verbal descriptions of tlie problefn situations (described in the foregoing) 
withi videotaped presentations of enacted and real-life interactions between 
CCs^ ^nd recruits. The computer, controls the videotape display, showing appro- 
priate segments of tape at appropriate times. A drill and practice mode of 
instruction is employed io teach t;he lessons contained in the videotape 
scenarios. . , 

.d. Video_FeedbacL Program. '- -In^^^^^ other- videotape- programi the com- 
puter provides automated information displays, tape control, arid data proc- 
essing as forms of assistance to "judges" in their efforts to describe and 
evaluate student CC performance in role-played situations and to feed back 
* thijs information to the students. 

20 

e. Attitudo Discrepancy Program.,, In the two Socratic approaches, the 
students are provided information and persuasive arguments in efforts to 
modify their behaviors. One such program determines discrepancies between a 
student's conceptions nnd intentions concerning the performance of- specisfic 
behaviors (e.g., seeking help from other CCsf) on the- one' hand, and rtc policy 
rcjgarding 'these behaviors on the other hand. The program then addresses any 
discrepancies that were found to exist by showing the \ student that it is in 
nis/her own best interest to comply with t|ie policy of^ RTC., 

f. Evaluation Information Program.'^' jhe second \Socratic approach re- 
veals any misconceptions which the student may have concerning the process 




weights are then compared with the actual weights employed at RTC (determined 
by having the actual evaluators evaluate the same hypothetical profiles). 
Any differences that occur between these two sets of weights denote miscon-i 
ceptions held by the student regarding the evaluation process. XCs are ex- , 
pected to obtain better evaluations when their understanding of the evaluation 
process becomes more accurate. 

g. Management and Evaluation Program. This program presents displays 
of data which evaluate the performance of CCs an(^ RTC in general. This 



^8. Seerfootnote I I on page 8. 

19. Well«r, D. R. and Blaiwes, A. S. Computer-^o.-o.^v 
of Interpersonal Skills, NAVTRAEQU IPCEN, in prepiiration. 

20. See footnotes 7 and 9 on .page 8. 

21. See footnotes 7 and 9 on page 8. 

22. See footnote 11 on page 8. 

11 



ssisted Judging and Feedback 



NAVTRAEQUIPCEN IH'307 ^' 

prpgram is used by CCs to* assess ^ diagnose , and improve their perfomiancei and 
by managers at RTC to identify areas that require action, to deterrtiine the 
particular action to be taken, and tb evaluate the effects of the action after 
it is taken. ., \ 

The major faatiJres of this program are: ■ 

a. Automatic storage and processing o t larq* ahuu • ^ on many 

critical aspects of the rtC operation (e.y. , o v ' i.,,: ^ ,(sures of re- 
cruit performance, attitudinal and evaluation uau- ..liuc ' I- surveys). 

' b. . Automatic computation of relationships and differences among all 
categories of these data. ' . ^ ^ ^ 



I c. Automatic di^p^Vays of these data with "i 
cant relationships contained in the\ data. 



solation of the nt)st signifi- 



d. Automatic /rankings of the CCs ir\ accordance with their performance on 
?ny system variables which are salected and i^ighted for importance hy RTC 
e valuators. ' 

DEMONSTRATE. AND EVALUATE COMPUTER-BASE&.PROG^R^^ There are two important as- 
pects to the demonstration and evaluation of the products of this project. . . 
They should'be: (1) f uncUanal ly effective such that they appropriately ' 
accomplish RTC functions, "and (2) cost-effective such that the RTC functions 
accomplished by the new programs justify the dost of thei^^ use. 

'Functional EffectiveneLSs - Pre FY^78. Of the seven programs described in the 
previous section, three >have been "evaluated in previous reports (the problem 
situation program, the attitude discrepancy program^ arid the eval uati on infor- 
mation program). The problem situation program was isel ected to be the basis 
of the current program. A^rief review of the previ'pus research on this 
program is presented below* For results of research^ on the other two programs 
see the reports referenced in the previous section. 

a. Skill Validity:' Previous studies have consistently demonstrated the ^ 
validity of the skills taught by the problem situation program. The skills . 
have been correlated with several outcome measures such as recruit morale, 
recruit inspection scores, and recruit retention in the Navy. Nearly all of 

"these correlations h^ave been po§.itive and significant, indicating that the . 
skills are related 'to important measures at' RTC. 

b. Experimental/Control Skill ^Differences: Another consistent result 
of the previous studies is that the experimental subjects (those taught with 
the program);.are rated by their recruits. as performing the skills to a greater 
degree than control subjects. This positive training effect is most pro- 
nounced in students who are more motivated to do the CC job. ' 

'■, "1:. Experimental /Control Outcome Differences: Results concerning experi- 
mental/control differences on outcome measures (inspection scores, morale, 
retention) have been less consistent. Experimental subjects have performed 
at a higher level in the majority of cases, but few differences have been 
significant. The obvious importance of the outconie measures would make it 
worthwhile to determine why positive results were obtained in some instances 
and r\qt in others. 
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Functional Effectiveness - FY 78. The program in use during FY 70 (GETS) was 
' based on the problem situation program, with the addition of the video situa- 
tion program and the management and evaluation program. The video feedback 
program is being evaluated and reported on in a separate effort. 

First, y^esu Its from the probl em si tuation and video situation programs 
are compared with th6 results of previous studies. Following this, an evalua- 
tion- of the management and j^evaluation program is presented. 

Probl em situation and vi^ ftuation programs: 

a. Ski ri_ Validity: it.^ , variables (e.g., skill performance, 
background, outcorries, etc vi?r' c lected on all CCs leading training units 
at RTC, Orlando from October i .)/ , until. August 1978 (a detailed description of 

^-all variableis is presented in a^ previous report ). Consistent with the pre- 
vious results, these data showed that the skills taifght in the two "situation" 
programs are valid. • Scores on the ten skill areas were correlated with re-r — 
^ruit inspection scores, recruit attrition scores, recruit morale scores, and 
division officer evaluation scores. These correlations are. presented in 
Table 1/ Of the 40 correlations presented, 34 are in the expected' direction. 
Of the 25 significant correlations, 23 are in the expected direction. 

. b. Experimental/Control Skill Differences: All subjects for the FY 78 
.study were recent graduaftes of Company Commander School at RTC, Orlando. Over 
a period of approximately six months., each class in CC school was randomly 
divided into control and exiJarimental subjects. The experimental subjects 
received the CETS training program in place of some "shadowing time" (i.e., 
watching active CCs do theiir jobs). This occurred :between graduation and the 
picking up of their first training uii-it. The control subjects received nc 
special tv^eutment; instead, they contir^ued to engage in normal shadowing 
activities. 

Data vjere Sbl lected for experimental and control subjects. The data were 
identified as coming from regular, experimental , or control CCs to facilitate 
statistical comparisons. 'Da^t^. were collected on a total of 36 experimental 
and 30 control subjects. > 

- iThe t-tests were performed on all skill variables as assessed by the re- 
cruit survey. These results are presented in Table 2 (numbers in Tables 2-7 
are mean value?). The N.'s for these and other tests exceed the number of sub- 
jects because many subjecfts.had lead more than one training unit, and data 
were' stored separately for §ach training unit. 

None of the t-values in Table 2.is sfgnificant, and the differences are- 
about equally divided bc^tween those favoring the controls and th^se favoring 
the experimental s. These results are inconsistent with the previous research. . 

A-poss^ble reason for these results concerns a policy change at RTC. 
•Coincident with the beginning'of the FY 78 study, RIC decided to place two CCs" 
in. cfiarge of each-training unit (one CC has lead each training unit in the 
past}.. Thus, each experimental an*d control subject was paired with another CC, 



23. See footnote 11 on page 8. 
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usually one who had lead^training units previously. It can be assumed that 
the second CC was a majo^ influence on the behavior of the subjects. This 
influence could easily be greater than the influence of the experimental treat- 
ment, and it would tend to decrease or eliminate any experimental effects 
(especially^where the training is .at odds with traditional methods of leading 
training units). 

■■)!-.. • ■ 
' c. Experimental/Contrpl Outcc^rne Differences: Tables 3 through 7 present 

the differences and t-values for .^He outcome variables. None of these differ- 
ences is significant. Recruit attitude scores are about evenly split between 
those favoring controls and thoVp f a vo ring experimental. The CC survey and 
MED scores predominant"^ - favc ontrols. The att ^ es all favor 

'e ekperimentPls, w > % ^t v' >^fficer survev . favor the 

i. jntrol s. ^. 

These results are disappointing, and it is reasonable to expect that the 
.addition of the second CC in a training unit would tend to reduct any experi- 
mental-effect on^theseva'riabl^^e^^^^ the ski 1 1 variables-. 

Managem'ent and evaluat-ion program. This program is considered separately 
from'the other/programs because it is not strictly a training program. It was 
originally designed for the collection and analysis of experimental /control 
data, but it soon became apparent that the program had potential as a general 
management/evaluation tool at RTC. . \. 

/ TABLE 1. SKILL VALIDtTY: CORRELATIONS WITH OUTCOMES ' 



SKILL 


RECRUIT 
MORALE 


RECRUIT 
AHRITION^ 


RECRUIT 
INSPECTION 


. D.O. • 

SURVEY ' 


Concrete 


,5^** 


: -^09 


.43** 


.16* • 


Timely 


.58** 


-.08 


.48** 


.27** 


Clarifying ,. 


.65** 


-.05- 


.22** 


.06 


Reasonable 


.04 


.20** 


-.09 


.14* 


. Relevant 


.74** . 


-.04 


.20** 


.06 ■ 


Considerate 


.64** 


r.l6* . 


-.09 


-.13* 


Human 


.66** 


-.07 


.05 


.00 


Goal Setting 


.53** 


-.03 


.40** 


.17*. 


-Instruction 


.67** 


-.11 


.34** . 


. .16*' 


Feedback 


.58** 


\ -.12* 


.21** 


-.02/ 


-j\ y~ ■ 


325 


' 263 


, m 


262. 



a. Negative correlations are desirable; i.e.,. high skill, perfonnance\r low 

attrition. ' . r 

■* Significant at .05 level. ■ /■ 

** Significant at ,01 level. /' 



SKILL 



Concrete 

Timely 

Clarifying 

Reasonabl e 

Relevant 

Considerate 

Human 

Goal Setting 

Instruction 

Feedback 

Counseling 

Motivating 

Reward/^isji^ 

Total Score 
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TABLE. 2. CC SKILL SCORES FROM RECRUIT SJRVEY 



EXPERIMENTAL 
(N=59) 



4.14 
4.14 
3.76 
3.07 
3.77 
3.55 
3.98 
4.05 
3.89 
3.62 
3.81 
3/.66 
3i57 



COITTROL 
(N=54 ) 



3.77 



4.21 
4.16 
3.77 
3.02 
3.79 
3.54 
3.96 
4.09 
3.96 

3./2 
3.65 
3.55 
3.77 



t- VALUE 



-.90 
-.36 
-.22 
1.21 
-.41 
.05 
.27 
-.88 
-.86 
-.03 
1.10 
.13 
.23 
,00 



TABLE 3. RECRUIT ATTITUDE SCORES 



AtTlTUDE 


EXPERIMENTAL 


COITTROL 




(N=59) 


(N=54) 


.' t-VALUE 


Respects CC 


4.38 


4.40 


-.18 - 


Morale 


3.89 


3.91 


-.23 


Boot Camp . 


2.85 


2.86 


-.21 


Esprit de Corps, 


4.47 


. V 4.43 


.'93 


Reenlistnient 


3.39 


3.36 


.58 


Navy 


4.32 ' 


4,31 


.43 


Traiyiing 


4.15 


4.11 


' .88 


Attitude Total 


3.92 


3.91 ,, 


.29 




TABLE 4. 


CC ATTITUDE SURVEY 






EXPERIMENTAL 


com-RQL 




ATTITUDE . 


(N=41) 


(11=38) 


t-VALUE 


CC School ; 


. 3.09 


3.38 


-1.74 


Job in General 


3.65 


3.93 


-1.61 


Recruits ' 


3.43 


3.57 


- .97 


Division staff 


3.92 


3.75 


1.01 


RTC . 


2.99 


'3.30 


-1.56 


Survey Total 


3.41 


3.59 


-1 .37 



15 



NAVTRAEflUIPCEN IH-307 





TABLE 5. DIVISION OFFICER SURVEY 






EX PER IMENTAL 


CONTROL 




VARIABLE 


(N=44) 


(N=44; 


t- VALUE 


Bearing 


4.27 


4.39 


- .83 


Reliability 


3.95 


4.14 


-1.42 


Adaptabil ity 


3.87 


4.05 


-1.47 


Initiative 


3.92 


4.14 


-1 .31 


Cooperativeness . 


4.01 


4.23 


-1.64 


Counsel ing 


3,85 


4.12 


-1 .52 


Management 


3.82 


3.94 


- .81 


Problem Solving 


3.80 


3.92 


- .88 


DisG^pldne 


3.85 


3.98 


- .87 


Quel ity V TU 


3.86 


'4.04 


-1.40 


Evaluation of CC 


3.29 ■ 


3.76 


T rt o 

\ -1.89 


* Survey Total \ 


3.B7 


4.06 / 


-1 .45 

— 


: . ■ \ . 

:■ \ 
\ 


TABLE 6. RECRUIT INSPECTION (MED) SCORES 




\ 

\ ■ 


EXPERIMENTAL 


CONTROL 




SCORE ."^ ' 


{N=41) 


(N=39) 


t- VALUE 


Personnel 


3.57 


3.62 


-1.30 


Barracks 


3.81 


3.84 


• - .72 


Locker 


3.07 


3.10 


^ rt 

- .38 >• 


infantry 


3.54 


3 .59 


-1.16 


Streetmarks , . 


.12.37 


12.38 


- .01 


RPBs ' 


88.4 . 


87.4 


.14 


MED Total • 


i 3.50 


3.54 


-1.00 




TA BL E 7. ' 


RECRUIT ATTRITION 






EXPERIMENTAL 


CONTROL 




VARIABLE 


.(N=42) . : 


(N=38) > 


t-VALUE 


Regular Setback 


6.40 


6;57 


-.17 


Special Setbacks 


13.95 


15.00 


-.87 


Re-recycles 


3'. 82 


4.36' 


-.80- 


Total Setbacks. 


20.20 


21.39 


-.63 



Basically^ the program allows rapid input, storage, processing, and output of 
a Wide variety of variables Concerning RTC. The processing options include 
grouping of data in a variety of ways: t-tesjs, analysis of variance, corre- 
lations, multiple regressions, and the automatic ranking of CCs according to 
their score*?, on a wide v^trlety of variables (selected ahd weighted by RTC 
personnel )c A more detailed description of this program is given In a pre- 

vious report. Examples of some of the outputs are given, in Appendix A. 



/^WI See footnote 11 on. page 8. 
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(For reasons of date- S':>curi^y» the data shown in this appendix are not actual, 
current data; but an fro.;, a develofinental test of the system, ) 

A program of this type has almost limitless applications, and RTC has 
already'used it in a variety of ways. The following items are a sample of 
some of the recent findings: 

£. Training \units with two CCs (as compared to those with one CC) have 
higher recruit inspection (MED) scores, higher skill performance ratings of 
CCs, higher division officer ratings of CCs, greater recruit respect for CCs, 
but greater attrition, 

b,. Recruits i^^ tr^^nlng unit with high MED scores are more f^v 
the current si tuatiuw (higher morale, respect for CC, liking of RTC)i but less- 
favorable about the jfuture (esprit de corps, attitude toward Navy in general, 
intention to reenlist), 

— — _G,— CCs who earn higher MEDs emphasize the technical side of this job at 
the expense of emotional considerations. 



ERIC 



d. Overall recruit attitudes are positively related to the overall, per- 
formance of the C£TS skills by their CCs; 

— • Division officers rate CCs higher when they tfend_±o_.emphas;l^ze the 
technical side of the job and deemphasize the emotional side. 

f« Total attrition is increasing at' RT€ over time. 

The programs have been used for other purposes at RTC, including the 
selection of the "sailor of the .quarter,/' the counseling of CCs, and the 
: Se/[ection of .perspnn^^ for certain jobs. Consideration is now being given Xo 
" the popibnity^^o^^^^^^^ CET? that would allow the prediction 

of the quality of a; CC's performance prior to and/or very early in his/her 
assignment as a CC (thereby providing a basis for selection and special early 
training for CCs), 

Cost-Effectiveness, The previous technical report from this pro ject^^ dis- 
■ cussed the cost-effectiveness of using the CETS program at RTCs, . The major 
conclusions fro'm this discussion were: , 

a. A major technicaland administrative effort would be required to 
" assess the costr-related effects of the program, 

b, A rational approach to organizational development requires cost- 
effectiveness data for all aspects of the ^operation of the RTCs, 

c* In the absence of fundamental effectiveness measures (i.e, , perform- 
ance of recruits after RTC), cost savings can be'assumed if the new programs 
provide more efficient methods of performing required tas|<s. The extent to 
which a task is required must be a subjective judgmen*^ at this time. The 

; 25. See footnote 11 on page 8. 
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tasks performed by the GETS system would appear to be required at RTCs, 
Clearly, the advantages of the GETS system over other methods of performing 
the same tasks are great, and easily justify the costs of the system, 

^ d. Measures of recruit attitude, morale, and motivation are difficult to 
quantify in fiscal terms; however, experimental results show that such data 
can be quite meaningful . These results indicate that these measures might be 
useful in reducing costs. 

e. Another approach to cost-effectiveness rissesi'^'^^nt ra.i ^ ^icve 
rrf<\^i programs aro <iubs;f»*^ ipfj for ex l5t programs. Unfortunately, most of. 
hhe pei w^d by urie GETS programs either v\ere not performed previously 

at RTGSy or a substitution approach to their evaluation was not feasible. The 
cost savings for the few areas where a substitution approach was used, however, 
indicate that these functions alone can justify the maintenance costs of the 
^systesin. Further, severaT additional areas where the system could easily sub- 
. . stitute for less efficient methods are apparent. Use of the system in these 
„ areas woul.drinaike^its. cost justifica^ evei more compelling. 

These conclusions still appear to be valid. A comprehensive cost-effec- 
^ tiveness assessment of . the GETS system continues to be elusive; however, addi- 
tional information that can be subjected to a conventional cost analysis has 
become available. 

* The purchase^ price of the system was $35,000; and $5,000 per year j's re- 
quired for maintenance. These costs are variable depending on when the system 
is acquired and which programs are^desired for implementation. For example, 
a computer system costing as little as $4,000 to purchase and $1,000 per ^year 
to maintain' would be adequate for operation of th6 basic problem situation 
training program. 

« ■ . > ' . , ' , ■ ■ ' . ' ' 

^^^^^^^^T T cost in a:ddition to equipment costs is the cost of the 

IDersonnel required for operation of the system.: This cost is also^ variable,^ 
, depending on the tasks assigned^ to the computer. .A minimum of about .12 man- 
Hours per week is required for clata collection and entry. However, . several 
hburs per week were devoted by RTG to survey administration before the advent. . 
0^ the^GETS system, so most of these 12 man-hours would be expended even with- 
out GETS. The number of additional man-hours required for other GETS related , 
tasks also depends on the extent to which each activity can be integrated ipto 
otner ongqing activities at RTG. For example, the cost of GETS related coun-. 
selving of CGs should not be attributed to GETS if the counseling replaces 
exis^^^^ at RTG. 

^ \ldealTy, an education/prograifrm billet should'be provided for the oper- 

ation and improvOT^^ of GETS. This, billet should free time from other billets 
in that it would accompl ish .tasks (e.g., cdtinseling, training, "^data analysis) 
that normal ly are performed by a number of other positions; and are performed 
less efficiently due to the lack of computer assistance. « % very difficult 
to ferret out the time savings that might result from the assignment of all 
\ such tasks, to a single GETS billet, but the :savings should be considerable. 
Furtheri\as more tasks are assigned to the computer in the future, the work 
load on dither RTG personnel should be further relieved. 
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The only quantifiable cost savings at this time are: (1) about $3,000 
par year for computer processing of recruit attitude data, and (2) about $3,000 
per, year for processing of CC school scores. No dollar estimates can be placed 
on the improved accuracy of the data provided. 

Proper use of the evaluation proqram H i ri r. it L(.i>!(i= -i.)- it 
savings. For example, it fs {jof-.ible • v .e. inii,.; ,i.:ch variables are related 
'■0 r..s.rw.'- attr tion. lising thu :.ystem, it has been found that CCs with five 
to 'fM years and over 20 years in the N&.vy have significantly greater total 
"drop out" rates (about 25%) than CCs with 0-5. years in the Navy (about 16%). 
Findings such as these suggest a potential for selecting and or traininvj CCs . 
in ways that lead to reduced attrition. Such possibilities would be pursued 
in. future efforts on the project. 

The CETS system should be used to determine the appropriateness of such 
actions, and .to monitor the effects of any actions th at are taken. A case of 
an actual application of this sort can be provided. The effects of operation- 
al change's at RTC are now determined in a small fraction of t he time that pr e- 

ties of the system. For example, the effects of changing from one CC to two 
CCs per training unit were determined in a couple of hours. This same task 
would have requi.-red at least two man months to accomplish prior to the intra- 
ductionof the system at RTC. 

Similar savings can be attributed to this system f^^^ host of other ad- ' 
mini strati ve type tasks. For example, if the process now being used to select' 
CCs for jobs and hopois (e.g., sailor of the quarter-) were to 'be attempted 
withoLit the aid of the system, an estimated 20 ..additional hours per week would 
. be required to even approximate the operations now performed. The system also 
produces information critical to the counseling of CCs, which would require 
untold hours to .acquire- by more traditional .means. (This is just a small ' 
. „ .samplje_pl_the_raany_-way_s-:-th — - — 
money saved due to these opef-ations, although considerable, is ill -defined at 
this time). . • ■ . -' . » - 

IMPLEMENT COMPUTER-BASED PROGRAMS. The CETS program has been in continuous 
use at RTCi Orlando for" approximately one year. RTC personnel have made ex- 
. tensive" use of the management and evaluation programs, and have expressed the 
desire to see these programs continue. Due.tb the disappointing results of " 
the recent evaluation of the training programs^ further development and testing 
would be required before the training portions of the CETS program are ready 
for o peratio nal acc eptance at RTC. ' „' „ : „; L „^ 

One of the concepts of the CETS evaluation and management programs is that 
. they pmvide built-in "self-correctional" features. Basically, CETS is de- 
signed to be used by RTC as a tool for organizational development, and it 
appears critical that the targets of this developent should include CETS 
itself. Thus f one appealing approach'to the enhancement of training in the 
area is to "prbvtrfe "s'el f-.corr^cting" capabilities (such as the present ones) 
wheraby the programs can be gradually "configured and. continually. reconfigured 
by operational personnelto meet operational requirements. - 

To help facilitate such applications of CETS in general areas of organi- 
zational management and development, a "performance report" was prepared based 
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on- the. data from GETS. Th1^: '-^1^. /! t,>> ag severe o ' imiteci ■ ' n n 
the subject, suir»i\4rized somt-- -^-h- i vrf, (see prrvir tion) md cc 

elusions de- f'^om the "^1.1$ ^ ^^''^ ^ d nanagemen' ;/steT and, on 
basis of these tendings, recommended actions for possible Implementation at 
RTC, Some examples of the actions recommended are: 

a. Promulgate the procedures employed with the GETS CC ranking program 
to evaluate and select CCs and extend the use of this ranking program to other 
areas at RTC, 

b. Establish the top ranked CCs and Divisions (as determined by GETS) as 
elite groups by recognizing and rewarding them in appropriate ways. 

c. Instruct CCs to adjust their instructional strategies to suit the re- 
quirements of particular instructiohal situations in accordance with findings 
from GETS (e.g., teaching infantry is facilitated by a "tougher" -approach - 
i.e., striving for higher goals and s.etting shorter time limits for learning - 
than in teaching other MED areas). - ^ - ^ — '- - 

d. Capitalize on the resources at RTC by soliciting lists of recommenda- 
tions for action from the%ost creative and capable personnel at RTC.. These 
recommendations would be based on these performance reports and other data 
that may be gleaned from GETS and other sources. Implement and evaluate (via 
GETS) 'the best recommendations and appropriately reward the people whp provide 
them. 

This first general performance report was intended to serve as a working 
model which could be-tmproved with use at RTC. One of the major benefits of 
the performance report would appear to be that it allows organizational errors 
to.be detected and openly acted upon. This is accomplished by bringing comfion 
in f >rmation on organizational functioning to t he atte ntion of ^arge numb er- 
of organizcitional managers. It seems that failures in organizational"" develop- 
men f efforts ^^ -V^^^a re often ignored *'>r denied*, sometimes remain' completely 
unrecognized, are rarely analyzed, and are almost never legitimized as v,aluable 
sources for learning. The result . . . is that errors in. (organizational 
development) . . . theory and technology are perpetuated, common problems are 
labeled unique, and dominant practices remain unquestioned.... We musjt learn 
how to reward the personal vision and organizational corranitment that fully 
recognize "the problems in creating change, that fully accept that errors are 
inevitable fh-these undertakings, yet still choose to risk, to persist, and .to 

-learn from the errprs . in .1 -(organijzational .development:)." Thus,- CETS/_ 

provides the basis for a critical but oftefi missing aspect of an organization, . 
viz., the ability to be "self-correcting." 

RTC needs to be able to develop and.use the performance report discussed 
in the foregoing, in order to optimally implement the "self -correcting " capa- 
bilities provided by GETS. A variety of procedures can be used to develbp the 
performance report, depending on the informational needs of RTC. The proced- 
ures employed /or the first prototype model of this report , were: . 



7^. Mirvis, P. H. and Berg, D. N. in ISR Newsletter. Institute for Social 
Research. The University of Michigan, Spring 1973, p. 6. 
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a. Ll 'delations were obtained between each variable of major interest 
and all othc variables ori the system, 

b. The t- tests and analysis of variance tests were obtained between and 
among all variables of major interest. . 

c. The significant relationships found with each test were noted and 
common findings among the various tests were determined. 

d. Tentative concllisions and recommended courses of action were derived 
based on clusters of these related findings. 

These procedures are quite preliminary. It is expected that 'improvements 
in the procedure would best evolve througfv-continued use at RTC. Efforts are 
now underway by RTC to secure support and funding to make the evaluation 
portion of GETS a permanent operational system at RTC. 

-^reJ4mna r^v^e^v idenGe - i s - a va i 1 abl p in i nH i r a tP _t.hfl±_BJr.^ _Tn-,t-hQ-iTi;^-inx^^ 

have the capabilities to properly use, maintain and extend the management and 
evaluation programs of GETS.. In the couple of. months since NAVTRAtQUIPGEN 
support has virtually^ended^ RTG, Orlando has.» finder their own initiatiye and 
control^ planned and iniH projects based on 

"GETS. Some of these projects which have come to the attention of the present 
authors are listed in the following: 

b Scheduling for^Master Training Schedule 

0 Plot variables over time by T. G. average' 

" 0 Gonvert DO surveys to annu-al evaluation marks 

~ ""o^""Gorrect'Tnd~P^f a school academic tests 

^ 0 Develop counselling problems on video-tape for GO School instruction 

. \\ . ' 

0 Use test-score program developed for the fourth item (above) to store, 
process and compute grades for GG school students 

0 ^Sort divisjori scores by specific divi5ion officer (by name) 

0- Gompare ASVAB scores (total and comporiient) with attrition 

0 Gompare AFQT and Navy screening test scores for recruits with recruit 
attrition 



0 Schedule, MED inspectipns 
0 Gompute and file the MED inspection results 
.0 . Maintain GG rotation list 

0 Put recruit test bank on the> computer and generate random tests for 
recruits quarterly (or other period) 
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0 Develop and test questionnaire items for selecting the best CC candi- 
dates for CC duty prior to their assignment at RTC. 

Trial and error is bound to occur in the process of developing and imple- 
menting improvements as suggested by these sorts of programs. However, due to 
the relative ease of detecting and correcting errors and generating new and 
innovative trials with GETS, these processes should be far less costly and 
more likely to succeed than if they were performed by the more traditional, 
manual methods. Thus, GETS can provide the means to greatly increase the 
speed and expand the scope of.:the trial and error process, which seems to be 
integral to the development and operation of new procedures (as represented 
in the foregoing) no matter who does it or hov^ it is done. RTG, Orlando has 
demonstrated an orientation and motivation at "sel f-correction" prior to the 
availability of GETS. These qualities are essential to the proper utilization 
of . GETS by an organization. Thus, RTG, Orlando currentl y pre /ides a desirable 
situation to test GETS. The extent to which this condition will be r;dintained 
at the RTG and can be produced j'n other organizations:, are topics for further 

=:reSe^t:rch:;==:=== : ^^ =rrzr r 



As an example of^a "trial" currently being conducted by RTC, a question- 
naire has .been, developed by RTG consisting of^ items which distinguished "good" 
andJ'pooriVXGs_ance„they are a A test is 
now being considered in which information on^' these same items will be obtaijied 
for candidate GGs prior, to their assignments at RTG. If these items distin- 
guish between good and poor GGs before tjie^^^^^ for GG- duty, GG - : 

' seiectton can" be dec^^^^^ firmer basi.s and the consequent elevation of GG 

quality could have a tremendous impact on recruit training. 

OTHER RESEARGH PRODUGTS WITH POTENTIAL FUTURE USES 

_^^J^his_rej;e^ci^ 

CCs at 'RTGs. However, the job of ^GG and. the' environment of RTG have much ir^ 
common with other jobs and other organizations. Thus, niany of the findings 
developed during the 'course of this project have rel evance for organizations 
in general, as^well as RTCs in particular.' Three areas, to which these findings 
apply.are: (a) leader/manager-training and evaluation, (b) the evaluation 
arid management of organizational operations, and (c) training media. 

a. Leader and manager training ancj/. evaluation. 

(1) -Inter peVsonal sk ills and/benaviors perf ormed by leaders were 



identified and relationships between this performance and the effectiveness of 
the leader were .establ ished. The val/idation of these skills arid behaviors, 
rWOUrMir_ecommendrz-tl)e4 performance-evaluation programs 

especially for situations similar to that of the GG. This information ^Iso 
contributes to the general literati/re on the charactef'istics' of successful and 
unsuccessful leaders. ,/ . 

(2') Gertain types of afttitudes held by leaders were, found' to be 
highly predictive. of their behavior, and other types of attitudes were found 
to be unrelated to their behavior. These, validated attitudes can provide in- 
structional objectives and c/iteria for training apd evaluation programs, as . 
well, as contributing scientific information on relationships between attitude 
and behavior. / 



ERIC 
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(3) Discrepancies in the conceptions held by the personnel in an 
organization concerning behaviors appropriate tc the organiza tiori, were identi- 
fied, quantified, and related to organizational position. This information 
should suggest instruction arid other actions designed to reduce the discrep- 
ancies existing in an organization. 

(4) A framework of computer-based procedures for effecting attitude 
and behavior change in leaders was developed.' The specifics of thes proced- 
ures were developed and tested, and organizational conditions that act to 
facilitate or inhibit these changes were noted. The seven basic approaches 
that make up this framework can be appl ied in total or in various combinations 
to achieve the training objectives and administer the evaluation programs 
identified in this project, as well as for additional purposes that may be 
devisedi This framework can also suggest aTterhate approaches to the training/ 
evaluation problem that might improve upon the current versions. 

(5) With certain procedural modifications, the materials of both 
the problem situation program and the Socratic programs can be applied with 
ncn-computer techniques. In fact, RTC has manually applied the behavioral and- 
evaluation criteria identified in the process of develof)ing the Socratic 
programs. Further, role playing exercises developed during the production of 
video tape lessons have been used by RTC on a routine basis before the asso- 
ciated cbmputer^programs were operational. 

" ^ (6) A job-oriented approach. to instruction (i.e., the instruction is 
related to specifics of- the CC's job) was demonstrated to be feasible for the 
leadership/management area. This is in. contrast to the more theoretical, ab- 
stract forms of instruction often used in this area. 

bp ^Management and evaluation of organizational operations. ^ 

- (1) Procedures and materials for involving managers in the use of 
scientific research methods as an integral part of their jobs were identified 
and developed, and their operational feasibility wa^s e stab! i shed. 

(2) Jhe degree to which valid and trusted measures of organizational, 
success are available was identified and the probable effects of deficiencies ' 
in these; measures wer;e discerned. The absence of . suitable success measures is 
a basie' problem for RTCs as y\/ell as other organizations j and R&D is needed to 
determine how to obtain and use them. . 

(3) The findings already derived f the evaluation program can be 
useOs-a^basfs'^fbr managerial actions at RTC for as long as the data remain • 
current. • Further data can be processed manually; however, if substantial data 
are required,^ this task would soon exceed the capabilities of the organization. 

c. ^Training media. - 

(1) The feasibil ity and desirabil ity of the coordinated uses of 
video tape and a computer in training was demonstrated. Instruction using 
life-like video tape situations was demonstrated, along with'methods for 
aiding in the evaluation 61; a person^'s interpersonal actions as seen on video 
tape .and the feedback of this evaluation to the performer. 

(2) A capability was developed for translating the TUTOR language of 
the PLATO IV system into a language compatible with NOVA minicomputers. ^ 
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SECTION III 
DISCUSSION AND CONCLUSIONS 

there is a sizeable body of research literature on the effectiveness of 
training and evaluation for the interpersonal and affective, areas of the 
leader/manager job. This research indicates that these training and evalua- 
tion programs often make little or no detectable difference to organizational 

success, or even to job behavior;. ' Still, the use of such programs is wide- 
sp-^ead, and seems to be increasing. Ttiat high importance is placed on Navy 
programs in this area is evident in the statements of people in the highest 
governmental positions, as well as in the nature, of ongoing R&D programs. For 
example, a recent message from Admiral Hayward, the newly appointed Chief of 
Naval Optirations (CNO), gave special emphasis to the need for improved manage- 
ment a nd 1 ea der shi p " One^ma jo r-CNO' ob jective^ w . _ - ----- 

through a program of formalized training in leadership and manage- 
ment skills at those stages in officers' and petty officers' ' 
careers where such training is most needed and can be especially 

productive." - 

^ Operational programs reflecting these concerns are also noted. For exam- 
ple,- a new BuPers program called Leadership and Management Education and 
Training (LMET)/is a ". . . centralized, systematic approach to leadership and 
management clevelopment . . ." which is intended to provide instruction for 

,". . . the whole Navy.i.. . ." ^ 

_ Thus^ it appeaj::s_that training and evaluation programs should and will be 
implemented in ■this area. Further ," these pro will be developed, Waluated 
and selected on some basis - intuitiv^, empirical, and/or experimental. AIT' 
such bases were employed in the present p&D. Taken as. a whole, the effective-, 
ness evaluations of the programs vere generally -favorable, especially when 
compared with results from similar efforts. However, the results were s^ll 
not strong enough to constitute a major contradiction to the generalization 
that program effectiveness in the area rests on sha key grounds. The major 
reason- for the lack of more compl ete confidence in the training programs is . 
the failure of the most recent experimental evaluation to support the favorable 
findings of the earlier evaUiations (more discuss-ion^of^program shortcomings is" 
contained inTt^he fbl low • Nevertheress," weral.l results fr6fn~the present 
project were sufficiently encouraging .to recpmijend further pursuit rof thjs R&D 
and impl emehtation of its, product^^^^^^^^ \ 



]^;^iedler,> F. E. TTien" rouble with Leadership Training is that It Does Not: • 

Train Leaders. Psychology Today, 'February 1973. v 

28 'The Centerline, Naval Training Equipment Center, Orlando, Florida. Vol., 

XI. No. 29, July 20, 1976. ! ' ' ^ 

29. Leadership arid Management Evaluation and Training (LMET). All Hands. 

May 1978. No. 736, pp' 4-7. 
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Rationale for these irscofimendations rests on the favorable nature of most 
of the findings of the experimental evaluations of the training programs.- 
Further, the face validitjy of the skills of these programs seems to be good 
(I.e., the skills being taught appear to represent the skills the Navy wishes 
to teach) and the methods for teaching the skills appear to be effective and 
efficient. The favorable face validity of the training technology enployed by 
GETS is contributed to by its use of automatic and individualized computer- 
assisted, features vrith the addition of computer- control led video tape preserta- 
tion?i air of which are integrated. into more general programs of performance 
-evaluation and data analysis. 

This favorable assessment efface, validity is further supported by the 
correlational data which demonstrated the empirical validity of the subject 
matter of the programs, and by the student and organizational acceptance of 
both the subject matter and procedures of the programs. 

Finally, the present programs provide sophisticated , field tested data 
collection and processing capabilities for use by managers in their organiza- 
tional development efforts, including the continual improvement of the training 
and evaluation systems. 

These accomplishments ar6 considerable. However, they do not obviate the 
shortcomings of the R&[H and these too must be addressed. Among the most 
apoarent and damaging of the shortcomings were the failures to more adequately 
(a) improve and evaluate the program materials; (b) obtain organizational 
emphasis on the use of the programs; (c) achieve organization-wide indoctrina- 
tion onto the programs; (d) implement the training in remedial and refresher 
modes. . 

it.,., 

Such deficiencies, however, seem to be natural by-products of the tradi- 
tional R&D approach, and are not likely to be remedied in the current effort 
(or in other efforts in the area) until alternate R&D paradigms are found. 
Perhaps the primary rationale for recommending further use of GETS at RTG and 
further development of the GETS concept in the mili'tary is that, with GETS, 
such alternative -R&D paradigms may have been found. The present programs 
apparently offer the only embodiment of certain alternative R&D procedures 
that seem to. be needed for improved resu"lts in the area. 

With traditional R&D procedures, a general program fomat is selected, 
often on intuitive or face-validity bases. Then, if research is performed. 
It essentially stops at the time the system is implemented in the operational 
environment. This approach would seem to work best where the continued 
validity of the instruction is-well assured. , In'cqntrast, in training for 
leadership and management, program validity is in doubt ihitially. and even 
more^ so in the future. . . ' / ., 

The need fbr.continuous evaluation, revision and update of training 
programs is beginning to receive some attention in the general training field. 
This, attention, however, is mainly in the form of proposals, speculatiijns and 
token 'gestures and very little seems to have been done to actually implement " 
a comprehensive program to facilitate these processes. Further, except for a 
few sporadic, enlightened R&D programs, this need still does not receive the 
emphasis it deserves. " 
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One example of such an enlightened program is the Instructional Systems 

Development program at NAVTRAEQUIPCEN. This project takes the position that 
instructor training courses will include . . a detailed *how to' of course . 
evaluation, revision and update," The authors state that • . this require- 
ment is considered of extreme importance if a viable weapons system training 
program is to be maintained,^' This requirement stems from changes in train injg 
Objectives, etc., as well as . . the normal tendency of a new program to ^ 
deteriorate with time and personnel turnover. • • • * 

.t 

To this end, this program has initiated an R&D program for providing com- 
puter aids to the process of developing and modifying course materials to be 
accomplished both before and after a training program has been implemented. 
The present technology, however, is designed to go beyond application to! 
single training systems. It is intended to- extend the self-correcting concept 
to aid in managerial efforts to maintain and improve entire organiz-etions^ 
which include all the individual training and operating systems that^ comprise 
the organization. 

What appears to be needed in the interpersonal skills area is an opera- 
tionally-attuned R&D program that continues as an integral and routine part of 
the operation of the system. This seems especially to be required in this 
area because the effects of the important variables of leader/manage^" perform- 
ance are accessible only in the ultimate operational context and over extended 
periods of time, , 

An illustration of how the traditional R&D paradigm fosters the sortScOf 
shortcomings noted in the foregoing is provided by the inability in this 
projj^ct to determine the influence on, trained CC performance. resul ting from the 
change from one to two CCs per training unit. With continuous^ R&D, enough 
data could be collected to ascertain the effects of the change; and with 
operationally-oriented R&D, only the situation where both CCs had the same 



traininq would need to be investigated (in 
trained]. 



present case, only one CC was 



Another problem encountered in the present R&D also indicated the need to 
deviate from traditional , short-term, non-operational R&D approaches. It is 
nearly impossible (if not undesirable) to isolate the control group from some 
aspect of the experimental treatment. Experimental contamination stemmed from 
actions taken by the RTC organization to transmit informati'on to anyone who 
needed it, even if that information .wa% part of the experimental treatment. 
Contamination probably also occurred in the course of casual interactions among 
experimental and control students. Such communications are desirable in that - 
they indicate positive attitudes toward the material s/and they assist in the 
instructional process. However, such actions defeat the purposes of the ex- 
perimental/.control approach, ' , 

30. Funaro, J. F, and Bird, R, G. US. Navy Instructional Systens Deveflopment 
Operational Aviation Programs, Interim Progress Report for US Air Force, 
Flying Training ISD Workshop^ Washington, DCo Human Factors Laboratory, 
NAVTRAEQUIPCEN, Orlando, FL,. 3 February 1976, 
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Further, proper implanentation of a program of this nature seems to re- 
quire universal (not experimental vs. control) administration. When all 
organizational personnel have been exposed to the same materials, the oppor- 
tunities for mutual reinforcanent of the instructibnal concepts and behaviors 
is greatest. With the present design, however, the control group and other 
members of the organization who were given a very superficial, if any, intro- 
duction to the programs were not in a position to contribute this reinforce- 
ment. On.the contrary, these unindoctrinated personnel would be expected to 
disparage new ways of acting that conflict with their own (such a reaction was 
often observed in students in their initial responses to training on GETS). 

> Not only should R&D be continuous with the training, but the training 
should also be continuous with each student. Permanent behavioral changes in 
this area apparently do not occur suddenly,, but rather undergo cycles of 
testing, rejection, and re-adoption by the student before they replace existing 
1 ifetime . habits and become stable components of a person's behavioral reper- 
toire. Thus, as with short-term R&D, it is not likely that short-term training 
in this area would demonstrate consistently large on-the-job effects. It is 
more reasonable to expect the desired outcomes to result frbm a series of 
short-term training experiences. . 

One prime functipn of the GETS programs is to provide diagnostic informa- 
tion on GG performance so as to provide a basis for continual ronedial training 
for the GG throughout his/her tour at RTG. Unfortunately, this function was 
developed too late to receive adequate evaluation. Primary among the advan- 
tages expected from the remedial training approach in this situation, is thbt 
it would"^llow GETS training to compete on an equal basis with the other cbn- 
^sider'able demands on a GG's time. Whereas GETS training is viewed as a means 
, for becoming a more effective GG, GGs have' other tasks to accomplish at RTG 
which are necessary if 'they are to operate at even a minimal level. Thus, 
these other tasks naturally take precedence over the "nice to know but not 
essential" perceptions of the GETS training. With the remedial approach, how- 
ever, GETS training would be instituted when a GG is failing to perform ade- 
quately on some aspect of the job which GETS. is designed to improve. This^ 
would change the status of the GETS training from "nice to know" to "essential" 
and greatly improve the fnotivation to learn. 

The introduction of remedial fy-aining is only one way that student motiva- 
tion could have been improved. As suggested in the foregoing, the failure of 
the orga-nization to introduce personnel to the contents of the programs, and 
to emphasize the use of the programs (which are attributed in large part to 
restrictions in the traditional R&D paradigm) could also undermine the motiva- 
tion of the students to learn and use the training"" material s. . Gorrection of 
the^e deficiences (which would be facilitated by GETS) should irake a major im- 
provement in training effectiveness. Evidence for the particular importance 
of student motivation in this area is. provided by the current finding of an 
interaction between student motivation and. training effectiveness (discussed 
in the foregoing) as v/ell as general confirmation of this finding in other re- 
search in the''area,"^l ' - 



TT, Hinrichs, J. Fersonnel Training in Handbook of Industrial and Organiza- 
tional Psychology. Rand McNalley, 1976, edited by.M. D. Dunnette, p^* 856. 
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Thus, un.der this plan, the design of the instructional and evaluation 
programs woul d be tentative and initially of secondary importance. The criti- 
cal part of the initial development wou1d.be to provide mechanisms suitable 
for use mainly by non-R&D personnel for continual modification and evaluation 
of theVprograms and an instructional framework into which the modifications 
could be readily implemented. GETS is intended to provide such capabilities. 

Only with the continuous information that a system such as GETS would pro- 
vide can programs be evaluated and more optimal training and evaluation con- 
figurations be designed. Further, the availability of continuous feedback is 
• important not only for the improvement of training and evaluation programs for 
leaders and. managers but it also -is critical to any manager's- effectiveness 
across a wide range of tasks and at all levels of an organization. 

The importance of feedback to managers and the problems involved in 
getting and giving it have been noted in the general literature on management 
and leadership processes, as addressed^ in the following: 

Human leaV-ning depends on feedback on the results of one's 
behavior. Given the fragmented, oral nature of managerial 
work, leaders often get vague and miscellaneous feedback. 
Because they , a re general i St s, superficially involved with 
many different projects, intimately involved with only a 
few, it is. difficult for them to give feedback as well. 
Both giving and receiving feedback, however, is critical 
to their effectiveness. ' ^' , 

Building a network of contacts, extracting important inr' 
formation and passing it or>, designing tasks to provide 
useful feedback, and simply^ exchanging information are 
' how managers spend the majority of their time... . 

Setting up feedback systems, judgment systems, and systems 

for dealing with stability and instsbil ity r*iay be a major 

factor in leaders/fip effectiveness. Such,. structures are ^ 

the leader's methods of chOjjping up reality and papturing 

vague intuition^ ar'^d bits of Information to see which 

• ■ 32 ■ 
a(^tions need, to be taken/ " > . - 

The evaluation progf-am of GETS is an extremely efficient mechanism for 
"building a network of con;tacts," "extracting important information," "ex- 
. changing information," and other functions that managers must perform .in order 
to obtain and provide the feedback required by j:heir jobs* To attempt to per- 
form these managerial functions by more traditional means would be exorbitant 
if not impossible. To fail to perform these functions' by any means would be to 
omit some essential elenents from the managerial process. 



1?^ Lombardo,'M. M. Cooking at Leadership: Some Neglected Areas. October 
.1977, Office of Naval Research TR pp, 19-20. 
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APPENDIX A . 

■ SAMPLES OF MANAGEMENT ANb EVALUATION SYSTEM OUTPUTS^^ 

(For reasons of data security, the data shown in this appendix are not actual, 
current data; but are from a developmental test of the system.) 

TOTAL N EQUALED 401, AND THE PRODUCT MOMENT CORRELATION WAS EQUAL TO 0.549. 

. Y = MORALE ' X = CONSIDERATE 

4.597-1 

4.538-1 • ' ■ . , 

4.479-1 XXX X X X X 

4.419-1 XXX'. XXX 

4.360-1 . X XX XXXX 

4.301-1 XX XXXX XXX 

4.241-1 . X XX XX XXXXX XXXX X Xo 

4.182-1 X XXXXXXX X XXXX X XXXX X X o 

4.123-1 X X XX 'X XXX XXXXX X XX X 

4.063-1 X XXX X XXXXX XXX X X XXX X 

4.004-1 ^ X X XX XXXXX XXX XX XXX X X 

3.945-1 X X'XXX X XX XXXXXX XXXX XX XXX 

3.885-1 X X .XXX XXXXXXXXXXXXXXXXX -X 

3.826-1 X XX X X XXXXX XXXX XXX X XXXX XXX X > 

3.767-1 X X X XXX XX XXXXXXXX XXX X X 

3.707-1 XX X XXXXXX X XX XX X XX X XX 

3.648-1 XXX X . X X X 

3.589-1 XX X X X X XX 

3.529-1 X X X XX XX XXX X X 

3.470-1 X X XX X X XX X X X X 

3.411-1 X . X XX 

3.351-1 0 X X X 

3.292-1 X X XXX X X 

3.233-1 XX X 
3.173-1 X 
3.114-1 X . X 

3.055-1 X •. . X ' . 

2.995-1 

2 . 936-1 X- 

2.877-1 X - X 
2.817-1 

2.758-1 ' . . 

2..^9M X. ' 

2.63y-l • ' . , - 

2.580-1 X , , 

1.832 - 2.400 , 2.96 7 3.535 4.103 4.670 

- 2.116 2.684. 3.251' 3.819 4.387 

The increment for X is 0.047; for' Y it is 0.059. , 

Figure A-1. Product -Mbment Correlation Output 
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CHOOSE AN OPTION AND A VARIABLE NUMBER 

VAR. VAR. VARfABLE • ASSIGNED 

# STATUS , NAME 

1. IN USE CONSIDERATE (27) 50% 

2. IN USE MORALE (36) 50% 



ranktnr 


COMPANY COMMANDER 


SrORF 


T U 'S LEAD 


TT 1 


AGl 


A 


4 516500 


? T U 'S 


#2 

IT tm 


. BTCS 


B 


4.457001 


1 T.U.'S 




AMSl 


C 


4 381500 


1 T.U.'S 


#4 


PHI 


D 


4 355500 


2 T.U.'S 


#5 


ABCS 


E 


4.354501 


1 T.U.'S 


#6 


; AE2 


F 


4.347000 


1 T.U.'S . 


J7 


BTC 


G 


4.326251 


2 T.U.'S 


#8 


HTC 


H 


4.316001 


1 T,;:.'s 


#9 


SK2 


I 


4.312751 


2 T.U.'S 


#10 


BMCS 


J 


4.311001 


1 T.U.'S ' 


#11 ' 


NCI 


K 


4.305000 


1 T.U.'S - 


#12 


YNl 


L . 


:4.301500 


1 T.U.'S 


#13 


ENCS 


M 


4.298000 


1 T.U.'S 


#14 


OSC 


N 


/ 4.291500 


1 T.U.'S 


#15 


MMCS 


0 


/f 4.268500 . 


.1 T.U.'S 


#16 


TM2 


? ■ 


\]) 4.266500 


1 T.U.'S : 


#17 


BTC 




:l 4.257251 . 


2 T.U.'i 


#18 


DKl 


R 


4.252501 , 


1 T.U.'S 


#19 


EMI 


S 


4.246000 


1 T.U.'S 


#20 


MMCS 


T 


4.241501 


1 T.U.'S 



Figure A-2. 



CC Ranking Program Output 
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CATEGORY VARIABLE BEING TESTED IS "MORALE" 

\ ■ • * 

- GROUP VARIABLE' t-TEST BEING PERFORMED ON IS ^'CC RANK" 



#04 


by #05; t 


= 


1.196 & df = 


58. 




by #06; t 




0.987 & df = 


149. 




by #07; t 




1.019 & df = 


156. 




by #08; t 




1.430 & df = 


28. 




by #09; t 


= 


1.374 & df = 


4. 


#05 


by #06; t 




-1 .633 & df = 


207 




by #07; t 


— 


-0.719 & df = 


214. 




by #08; t 




0.397 &,df = 


86. 




by #09; t 




0.031 & df = 


62. 


#06 


by #07; t 




1.120 & df = 


305. 




by #08; t 




1.712 & df = 


177. 




by #09; t 




0.592 & df = 


153. 


#07 


by #08; t 




0.960 & df = 


184. 




by #09; t 




0.269 & df = 


160. 



#08 by #09; t = -0.171 & df = 32, 

MEAN FOR CC RANK = E4 WAS '4.2530 
MEAN FOR CC RANK = E5 WAS 3.8856 
MEAN FOR CC RANK- E6 WAS 3.9604 
MEAN FOR CC RANK =^ E7 WAS 3.9207 
MEAN FOR CC RANK = E8 WAS 3.8591 
MEAN FOR CC RANK = E9-WAS 3.8812 

ANOVA TABLE BY CC RANK FOR MORALE 



SOURCE ss df MEAN SQUARE F. 

TOTAL 37.699 401. 

BETWEEN 0.523 5. 0.105 1.112 

WITHIN 37.176 395. 0.094 

Figure A-3, t-Test/Analysis of Variance Output 
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NAVTRAEQUIPCEN IH-307 
YOU HAVE CHOSEN "MORALE" AS THE Qt^PENDENT VARIABLE 



1) . 

2) . 

3) . 

4) . 

11: 

■ XX 7 . 
8). 

10). 
XX 11). 

12). 
XX 13). 

14) . 

15) . 

16) . 

17) . 

18) . 
. 19). 

20) . 

21) . 

22) . 



TYPE IN THE NUMBER NEXT TO THE INDEPENDENT VARIABLE 
TO INCLUDE IN/OaETE FROM THE REGRESSION ANALYSIS, /OR 
TYPE 'WTO RETURN TO THE LIST OF DEPENDENT VARIABLES, /OR 
TYPE 'N' TO SEE THE NEXT PAGE OF CORRELATIONS, /OR 
TYPE 'P' TO SEE THE PREVIOUS PAGE OF CORRELATIONS 



MORALE 


R 




1.000 


(N 




325.) 


RECRUIT AHITUDES 


R 




0.824 


(N 




325.) 


MOTIVATING 


R 




0.745 


(N 


= 


325.) 


RESPECTS CC 


R 


= 


0.677 


(N 


= 


325.) 


INSTRUCTION 


R 


' = 


0.669 


(N 


= 


325.) 


REWARD & PUNISHMENT 


R 




0.665 


(N 


= 


325 . ) 


COUNSELING (RC) 


R 


= 


0.657 


(N 


= 


325.) 


CLARIFYING 


R 


= 


0.653 


(N 




325.) 


RELEVANT 


R 




0.631 


(N 




325.) 


BOOT CAMP . 


R 




0.614 


(N 




325.) 


CONCRETE 


R 


= 


0.583 


(N 




325.) 


TIMELY 


R 


= 


0.583 


(N 




325.) 


CONSIDERATE 


R 


= 


0.581 


(N 




325.) 


HUMAN 


R 




0.580 


N 




325. 


FEEDBACK 


R 




0.574 


(N 




325.) 


TRAINING 
NAVY 


R 




0.555 


(N 




,325.) 


R 




0.538 


(N 




325.) 


ESPRIT DE CORPS 


R 




0.533 


(N 




325.) 


GOAL SETTING 


R 




0.496 


(N 




325.) 


RE-ENLIST 


R 




0.360. 


(N 




325. ) 


T. GROUP 


R 




0.187 


(N 




325 . ) 


MED PERSONNEL 


R 




0.1 62 


(N 




319.) 



Figure A-4. Multiple Correlation Output 



NAVTRAEQUIPCEN IH-307 ' 

MULTIPLE REGRESSION FOR MORALE 

with 

CONCRETE 
CONSIDERATE 
COUNSELING (RC) 

REGRESSION STATISTICS: " 

MULTIPLE R = 0.69558 
R SQUARED < = 0.48383 
STANDARD ERROR = 0.33478 

ANALYSIS OF VARIANCE SS OF MS F 

REGRESSION 33.72176 3. 11.24059 100.29560 

RESIDUAL 35.97595 .321. 0.11207 

REGRESSION "EQUATION IS: 

MORALE = -0.67244 (CONSTANT) 

+ CONCRETE X 0.34632 (BETA = 0.27406) 

■+ CONSIDERATE X 0.39555 (BETA = 0.22378) 

■ ' + COUNSELING (RC) X 0.41451 (BETA = 0.29551 ) 

Figure A-5. Multiple Regression Output 
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DATA SELECTED FROM COMPANy COMMANDER 3JRVEy. PLOT CATEGORY IS: "JOB IN GENERAL" 
PLOT BEGINS WITH T.U. 13001 AND ENDS WITH T.U. '13060 FOR DIVISION #7 

~\ ' 2 T~ 4 1 VARIABLE VALUE 



mmmmmmmnmmmmmmmmmmmmm 5.000 December 
mnmmmmnmmmmmmmmmmim 4.333 
mmimmmnmmMmmmmmmnmmmmm 5.000 
mmmmmmmmmimmm ' 3.000 ■ January 

mmmmmmMimmmmmmmmm 4.000 
mmmmmmmmmmmmmmmnm 4.000 

. mnmmmmmmmmmmnmmmmm . - 4.333 februaky- 
mnmimmmmmmnmmmmmm 4.000 march 

mmmmmmmimmmmmimmm m 
mimmmmmmmmmmmmimm m ' 

mmmmmmmmmmmmmmmmmnmm m 
mmmmmmmmtmmmmmmmmimtim 4.666 april 
immmmmmmmmmm, 2.666 
mmmmmmmmmmmmmmnmm 4.000 
mmmmmmmmmmmmmimmmim 4.333 

mmmmimmmmmmm ' 2.666 may 
mmmnmmmmmmmmmmmm m 
mmmnmmmmmmmmmmmmi 4.000 june 

mmmmmmmmimmmmmmm m : 

mmmmmmmmmmmmmmmmim ■ 4.333 
mmnmmmmmMmmmmmimmm 4.333 july 

mmmmmmimmmmmmmmmti^ 4.333 
immmmmmmmmmmmmmmmm . 4.333 

mmmmmmmmmMiimtmimimmm . 4.333 
mmmmmmmmmmmmmmmmmnnm w august 

mmmmmmmmnmmmimmmm ^ m 

mmmmmmmmmmimmmmmmmm , 4.500 

mmmmmmmmmmmmmmmi^ ■ / 4,083 

mmmmnmmmmmmmmmmmmmm -^ 4.500 

mnmmmmmmmmmmmmmmmmmmm , 5.000 September 

mimmmmmmmmmnmmmmmmimif (4.667 . 

"average this plot = 4.137 N = 31 

' Figure A-6, Historical Plot of . Variable ' ' 



